A mini python based tool designed to convert various types of files and GitHub repositories into LLM-ready Markdown documents with metadata, table of contents, and consistent heading styles. Supports multiple file types, handles zip files, and has GitHub integration.
A post discussing new techniques developed for parsing and searching PDFs, focusing on turning them into a hierarchical structure for RAG search. The approach involves dynamically generating chunks for searches, sending headers and sub-headers to the Language Model along with relevant chunks.
The llmsherpa project provides APIs to accelerate Large Language Model (LLM) projects. It includes features like LayoutPDFReader for PDF text parsing, smart chunking for vector search and Retrieval Augmented Generation, and table analysis. It is open-sourced under Apache 2.0 license.
A collection of RAG techniques to help you develop your RAG app into something robust that will last
The article proposes a new framework, LongRAG, that aims to improve the performance of Retrieval-Augmented Generation (RAG) by using long retriever and reader components. LongRAG processes Wikipedia into larger 4K-token units, reducing the total units from 22M to 600K, thus decreasing the burden on the retriever. The top-k retrieved units (≈30K tokens) are then fed to a long-context Language Model for zero-shot answer extraction. LongRAG achieves EM of 62.7% on NQ and 64.3% on HotpotQA (full-wiki), which is on par with the state-of-the-art model.
Learn about the LLM Knowledge Graph Builder, an online tool that uses machine learning models to transform unstructured data into a knowledge graph. This tool is integrated with a Retrieval-Augmented Generation (RAG) chatbot and is part of Neo4j's GraphRAG Ecosystem Tools.
Learn about methods to ground LMs facts with examples from text using retrieval augmentation, generating factual answers and reducing hallucinations of super intelligence from AI.
This article explains Retrieval Augmented Generation (RAG), a method to reduce the risk of hallucinations in Large Language Models (LLMs) by limiting the context in which they generate answers. RAG is demonstrated using txtai, an open-source embeddings database for semantic search, LLM orchestration, and language model workflows.
pgai brings AI workflows to your PostgreSQL database. It simplifies the process of building search and Retrieval Augmented Generation (RAG) AI applications with PostgreSQL by bringing embedding and generation AI models closer to the database.
This article guides you through the process of building a local RAG (Retrieval-Augmented Generation) system using Llama 3, Ollama for model management, and LlamaIndex as the RAG framework. The tutorial demonstrates how to get a basic local RAG system up and running with just a few lines of code.
